Home ui system for managing digital ink

ABSTRACT

A home user interface (UI) system for managing digital ink is provided. The home UI system includes a plurality of state sensors capable of detecting a state in a house or a change in the state, and a plurality of home-use devices that are provided in the house or that form part of the house and that each include a handwriting sensor capable of detecting handwriting made by a person. The home UI system further includes one or more notification units that are configured to notify the person of the existence of the home-use device or a detectable region of the handwriting sensor of the home-use device. The home UI system further includes a controller which, when it is determined that a notification is necessary from a detection result of one or more of the state sensors, instructs at least one of the notification units to carry out the notification.

BACKGROUND Technical Field

The present disclosure relates to a home user interface (UI) system formanaging digital ink.

Description of the Related Art

Digital ink generated from an electronic pen is data used to reproducethe trace made by the electronic pen, which is similar to the trace lefton paper by a conventional pen used to perform handwriting. An exampleof digital ink that is an object-based data model residing in a computeris disclosed in U.S. Pat. No. 7,158,675, and an example of aserialization format used for digital ink is disclosed in U.S. Pat. No.7,397,949.

Another type of digital ink data is known that goes beyond merelyreproducing handwritten traces and that enables recording of “when, bywhom, where, and in what situation” the handwriting has been performedby a person to leave the trace. For example, Japanese Patent No. 5886487discloses digital ink that makes it possible to identify who has writtenstroke data that represents the trace. US Patent Application PublicationNo. 2016/0224239 discloses digital ink that allows acquisition ofinformation as context data when stroke data is input, such as theauthor information, pen identification (ID), clock time information, andlocation information acquired by the global positioning system (GPS),and recording it as metadata.

In recent years, studies have been made to use digital ink to estimate“with what kind of thought or emotion a handwriting has been made.” Forexample, in Japanese Patent Laid-open No. 2010-131280, a method and adevice are disclosed that ascertain and determine the mental,psychological, and physiological states of the writing person byquantifying time-space information of handwritten characters or picturesand extracting characteristic features. In International PatentPublication No. WO 2018/043061, an emotion estimation system isdisclosed that associates the writing state, such as the writingpressure, with biological information corresponding to emotion, andderives the biological information corresponding to the emotion onlyfrom the writing state.

Such digital ink indicating “when, where, and how handwriting has beenmade” can be considered a trace of a writing person, i.e., history dataof the person's action or emotion. Such digital ink can be combined witha personalized artificial intelligence (AI) to support realization of amore advanced life style.

For example, in US Patent Application Publication No. 2016/0259308, asmart home automation system is disclosed that senses an environment ina house or a person's motion and dynamically changes, based on thesensing detection result, a “policy” that characterizes operations ofrespective home-use devices.

BRIEF SUMMARY

By incorporating a handwriting sensor into a home-use device that isprovided in a house or that forms part of the house, the “house itself”can be configured to function as an input-output device of digital ink.However, when each of multiple members of the house can access pluralhome-use devices, multiple home-use devices or multiple writing personsmay coexist at the same time of digital ink input, and multiple piecesof written information may coexist at the same time of digital inkoutput. Thus, the following operational inconveniences may occur, forexample: [1] a use opportunity may be lost when a home-use device isoverlooked; [2] the identify of a writing person cannot be determinedwhen one electronic pen is shared by multiple people; and [3]handwriting information fragmentally input from multiple home-usedevices cannot be effectively utilized.

An aspect of the present disclosure is directed to providing a home UIsystem for managing digital ink, when a house itself is configured tofunction as an input-output device of digital ink, wherein the home UIsystem is capable of suppressing inconveniences due to multiple home-usedevices and multiple writing persons coexisting at the same time ofdigital ink input, or effectively outputting handwriting informationindicated by the digital ink.

A home UI system for managing digital ink in a first aspect of thepresent disclosure includes a plurality of state sensors capable ofdetecting a state in a house or a change in the state, a plurality ofhome-use devices that are provided in the house or that form part of thehouse and that each include a handwriting sensor capable of detectinghandwriting by a person, one or more notification units that notify theperson of existence of the home-use device or a detectable region of thehandwriting sensor, and a controller that, responsive to a determinationbased on a detection result from one or more of the plurality of statesensors that a notification is necessary, instructs at least one of thenotification units to perform the notification.

A home UI system for managing digital ink in a second aspect of thepresent disclosure includes a plurality of state sensors capable ofdetecting a state in a house or a change in the state, a home-use devicethat is provided in the house or that forms part of the house and thatincludes a handwriting sensor capable of detecting handwriting by aperson, and an estimation unit that, responsive to a detection by thehome-use device of a handwriting input, estimates a person who hasinputted the handwriting based on a detection result from one or more ofthe plurality of state sensors.

A home UI system for managing digital ink in a third aspect of thepresent disclosure includes a plurality of home-use devices that areprovided in a house or that form part of the house and that each includea handwriting sensor capable of inputting handwriting by a person, astorage device that stores handwriting data indicating a form ofhandwriting in such a manner as to associate the handwriting data withthe home-use device, a terminal device configured to display an image ora video and associated with one or more of the plurality of home-usedevices, and a controller that, in response to receiving a predeterminedoperation through the terminal device, acquires handwriting data of thehome-use device associated with the terminal device from the storagedevice, and instructs the terminal device to display written-by-handinformation indicated by the acquired handwriting data or contentinformation specified by the acquired handwriting data.

A home UI system for managing digital ink in a fourth aspect of thepresent disclosure includes a plurality of state sensors capable ofdetecting a state in a house or a change in the state, a home-use devicethat is provided in the house or that forms part of the house and thatincludes a touch panel display capable of inputting and outputtinghandwriting by a person, a storage device that stores handwriting dataindicating a form of handwriting in such a manner as to associate thehandwriting data with the home-use device, and a controller that, inresponse to one or more of the plurality of state sensors detecting apredetermined state of a person who exists near the home-use device,acquires handwriting data associated with the home-use device from thestorage device, and instructs the home-use device to displaywritten-by-hand information indicated by the acquired handwriting data.

According to the first aspect of the present disclosure, theinconvenience due to coexistence of multiple home-use devices at thetime of digital ink input can be avoided.

According to the second aspect of the present disclosure, theinconvenience due to coexistence of multiple writing persons at the timeof digital ink input can be avoided.

According to the third and fourth aspects of the present disclosure, thehandwriting information indicated by digital ink can be effectivelyoutput.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is an overall configuration diagram of a home UI system in oneembodiment of the present disclosure;

FIG. 2 is a diagram illustrating one example of a partial layout of ahouse in FIG. 1;

FIG. 3A is a diagram illustrating one example of a data structure ofsensor information in an environmental information database (DB);

FIG. 3B is a diagram illustrating one example of a data structure offirst device information in the environmental information DB;

FIG. 3C is a diagram illustrating one example of a data structure ofsecond device information in the environmental information DB;

FIG. 4 is a flowchart of a first operation of the home UI system;

FIG. 5 is a diagram illustrating the configuration of a determinationimplement included in a data analyzing section of FIG. 1;

FIG. 6A is a diagram illustrating one example of notification by ahome-use device;

FIG. 6B is a diagram illustrating the one example of notification by thehome-use device;

FIG. 6C is a diagram illustrating the one example of notification by thehome-use device;

FIG. 7 is a flowchart of a second operation of the home UI system;

FIG. 8 is a diagram illustrating the configuration of a discriminationimplement included in the data analyzing section of FIG. 1;

FIG. 9 is a flowchart of a third operation of the home UI system;

FIG. 10A is a diagram illustrating one example of a display on aterminal device;

FIG. 10B is a diagram illustrating one example of a display on theterminal device;

FIG. 11 is a flowchart of a fourth operation of the home UI system;

FIG. 12A is a diagram illustrating a first example of a display on thehome-use device;

FIG. 12B is a diagram illustrating a first example of a display on thehome-use device;

FIG. 12C is a diagram illustrating a first example of a display on thehome-use device;

FIG. 13A is a diagram illustrating a second example of a display on thehome-use device;

FIG. 13B is a diagram illustrating a second example of a display on thehome-use device; and

FIG. 13C is a diagram illustrating a second example of a display on thehome-use device.

DETAILED DESCRIPTION

A home UI system that manages digital ink in the present disclosure willbe described with reference to the accompanying drawings.

Configuration of Home UI System 10

Overall Configuration FIG. 1 is an overall configuration diagram of ahome UI system 10 in one embodiment of the present disclosure. The homeUI system 10 is configured to be capable of providing a “ubiquitous inkservice” which supports input and output operations of handwritinginformation at various locations in a house 100. The home UI system 10may be implemented not only in the house 100 but in various otherprivate spaces. Specifically, the home UI system 10 may be configured toinclude a digital ink server 12, a storage device 14, an Internet ofThings (IoT) hub 16, plural state sensors 18, plural home-use devices 20and 22, and a terminal device 24.

The digital ink server 12 is a computer that carries out comprehensivecontrol relating to input and output of digital ink, and is specificallyconfigured to include a communication section 26, a control section 28,and a storing section 30.

The communication section 26 is an interface that transmits and receivesan electrical signal to and from an external device. The control section28 comprises an arithmetic processing device including a centralprocessing unit (CPU) and a graphics processing unit (GPU). The controlsection 28 functions as a data acquiring section 32, a data analyzingsection 34, and a device control section 36 (controller), by reading outa program stored in the storing section 30 and executing the program.

The storing section 30 comprises a non-transitory, computer-readablestorage medium. Here, the computer-readable storage medium is a portablemedium such as a magneto-optical disc, a read-only memory (ROM), acompact disc (CD)-ROM, or flash memory or a storage device such as ahard disk incorporated in a computer system.

The storage device 14 comprises a hard disk drive (HDD) or a solid statedevice (SSD), for example, and stores various pieces of data to behandled by the digital ink server 12. Specifically, the storage device14 includes a database that accumulates handwriting informationgenerated in the life within the house 100 (hereinafter, handwritinginformation DB 38) and a database that accumulates environmentalinformation of the house 100 (hereinafter, environmental information DB40).

The IoT hub 16 is a relay equipment that can bi-directionallycommunicate with each piece of IoT equipment that exists in the house100. This allows each piece of IoT equipment to supply its own data tothe digital ink server 12 through the IoT hub 16 and a network NW. Thedigital ink server 12 can instruct each piece of IoT equipment to carryout any necessary operation through the network NW and the IoT hub 16.In the example of this diagram, the IoT equipment includes the statesensor 18, the home-use device 20 or 22, and the terminal device 24.

The state sensor 18 is a sensor that detects a state in the house 100 ora change in the state. The state in the house 100 may be an internalstate of the house 100 or may be the state of a person Hm who is presentin the house 100. The internal state of the house 100 may include anenvironmental state such as the state of air, light, and sound, thearrangement or use state of the home-use devices 20 and 22, and soforth. The state of person Hm may include the position, posture, motion,physical condition, emotion, and so forth of person Hm.

Various kinds of state sensors 18 may be used according to the detectiontarget, the room layout of the house 100, the installation cost, and soforth. For example, the state sensors 18 may be non-contact-type sensorsincluding a camera, an illuminometer, a thermo-hygrometer, a soundsensor, an ultrasonic sensor, a motion sensor, and an infrared sensor,and may be contact-type sensors including a pressure sensor, a touchsensor, and a motion sensor. The state sensors 18 may be stationary-typesensors arranged in the house 100 and may be portable-type sensors(which may additionally be wearable-type or non-wearable-type) broughtinto the house 100.

The home-use devices 20 and 22 are devices that are provided in thehouse 100 or that form part of the house 100. As examples of the former,home appliances (including a terminal device to be described later),furniture, home accessories and equipment, and so forth may be providedas the home-use devices. As examples of the latter, the home-use devicesmay form a wall, a floor, a window, a pillar, and so forth, of the house100. The home-use devices 20 and 22 are each configured to include amain unit 50 for executing its own functions and a handwriting sensor 52that can detect handwriting made by person Hm. The handwriting sensor 52is configured to include plural sensor electrodes for detecting theposition of an electronic pen 54 or a finger. The electronic pen 54 is,for example, a stylus of an active capacitive system (AES) or anelectromagnetic induction system (EMR).

The home-use device 22 further includes an indicator 56, in addition tothe main unit 50 and the handwriting sensor 52, differently from theother home-use device 20. The indicator 56 is an indicating instrumentthat can output information that appeals to the five senses of personHm, and may comprise a lamp, a speaker, a vibrator, or a display, forexample. This allows the home-use device 22 to indicate its ownexistence or its detectable region.

The terminal device 24 may be a multi-function, multi-purpose device,such as a personal computer, a tablet, a smartphone, and an AI speaker,for example. The terminal device 24 may be a stationary device or may bea portable device. Hereinafter, devices that can inform person Hmthrough output of information will be collectively referred to as“notification devices 58” in some cases (the home-use device 22 and theterminal device 24 in the example of FIG. 1).

FIG. 2 is a diagram illustrating one example of a partial layout of thehouse 100 in FIG. 1. In a kitchen 102 of the house 100, a camera as thestate sensor 18 and a window as the home-use device 20 are provided. Ina bathroom 104 of the house 100, a bathroom scale as the state sensor 18and a wall as the home-use device 22 are provided. Furthermore, personHm who is a living member of the house 100 possesses a smartphone as theterminal device 24.

FIG. 3A to FIG. 3C are diagrams illustrating one example of datastructures of the environmental information DB 40 in FIG. 1. Morespecifically, FIG. 3A illustrates sensor information 42, and FIG. 3Billustrates first device information 44, and FIG. 3C illustrates seconddevice information 46. In the environmental information DB 40, map dataindicating the room layout of each house 100 is also stored together.

As illustrated in FIG. 3A, the sensor information 42 is table data thatindicates the correspondence relation among “sensor ID” that isidentification information of the state sensor 18, “sensor type” thatindicates the type, “position information” that indicates the location,and “range information” that indicates the detectable range. Theposition information may be, for example, a floor, a room, acompartment, or relative coordinates from absolute coordinates (forexample, latitude and longitude) or a reference point. The rangeinformation may include various kinds of information that can define thedetection range. For example, the range information may be geometricinformation including the coordinates of a characteristic locationpoint, the length of a boundary line, the radius, and the angle range,or may be the name of a compartment or a section (for example, a kitchenor a bathroom).

As illustrated in FIG. 3B, the first device information 44 is table datathat indicates the correspondence relation among “first device ID” thatis identification information of the home-use devices 20 and 22, “devicetype” that indicates the type, “position information” that indicates thelocation, and “range information” that indicates the accessible range.

Here, the “accessible range” means the range in which person Hm presentin the range can visually recognize the home-use device 20 or 22 or therange in which person Hm in the range can touch the home-use device 20or 22.

As illustrated in FIG. 3C, the second device information 46 is tabledata that indicates the correspondence relation among “second device ID”that is identification information of the notification device 58,“device type” that indicates the type, “user ID” that is identificationinformation of the owner, and “first device ID” of the home-use devices20 and 22 associated with the notification device 58. For example, whenthe notification device 58 is the home-use device 22 (including theindicator 56), this home-use device 22 may be designated as thenotification device 58. On the other hand, when the notification device58 is the home-use device 20 (without the indicator 56), the terminaldevice 24 may be designated as the notification device 58.

First Operation

The foregoing has described the configuration of the home UI system 10in the present embodiment. The following describes operations of thehome UI system 10 including a first, second, third, and fourthoperations.

When the handwriting sensor 52 is incorporated into the home-use device20 or 22, to improve the appearance of a product, the home-use device 20or 22 may be designed such that the form of the handwriting sensor 52 isinconspicuous. In the case of arranging the home-use device 20 or 22 ina well-blended manner in the house 100, due to the camouflage effect (orthe stealth effect), it may become difficult for a user to determine theexistence of the handwriting sensor 52 or the position of the detectableregion that blends in the ambient environment. A similar phenomenon mayalso occur when the inside of the house 100 is dark, or when thehome-use device 20 or 22 is misarranged so that its detection surface ishidden.

Thus, the home UI system 10 carries out the “first operation” to timelynotify the location of the home-use device 20 or 22 or the detectablerange of the handwriting sensor 52 to a user who may be present. Thefirst operation will be described below with reference to a flowchart ofFIG. 4 and FIGS. 5-6C.

Flow of First Operation

In a step S11 in FIG. 4, the data acquiring section 32 of the digitalink server 12 acquires data that indicates a state in the house 100 or achange in the state (hereinafter, referred to also as “state data”) fromone or plural state sensors 18.

In a step S12, the data analyzing section 34 carries out primarydetermination relating to whether or not a notification to be describedlater is necessary, by using the state data acquired in the step S11.Specifically, the data analyzing section 34 analyzes the state data frommultiple perspectives or in a time-series manner by using variousanalytical methods including sensor fusion, and checks whether or not apredetermined motion by person Hm has been detected. Here, the“predetermined motion” means a motion that is a precursor of a writingoperation on the home-use device 20 or 22. For example, a predeterminedmotion may be [1] a motion of approaching the front of a refrigeratorand stopping there, or [2] a motion of stepping on a bathroom scale. Theformer corresponds to a “precursor” of a series of actions in whichperson Hm opens a door of the refrigerator, checks the content therein,closes the door of the refrigerator, and makes a note of food items thatneed to be additionally bought. The latter corresponds to a “precursor”of a series of actions in which a person gets on the bathroom scale,checks the measurement result, steps off the bathroom scale, and makes anote of the measurement result.

In a step S13, the data analyzing section 34 checks whether or not thedetermination result in the step S12 satisfies a primary determinationcondition. When the determination condition is not satisfied (step S13:NO), the operation returns to the step S11 and the steps S11 to S13 aresequentially repeated until the condition is satisfied. On the otherhand, when the determination condition is satisfied (step S13: YES), theoperation proceeds to the next step S14.

In the step S14, the data analyzing section 34 calculates pluralfeatures used for secondary determination to be described later from thestate data acquired in the step S11 or state data additionally acquired.For example, when the state sensor 18 is a camera, “physical features”may be extracted including characteristics of the face, the height, thebody shape, and the clothes of the writing person, or “motion features”may be extracted including the writing person's dominant hand and habitsof writing. When the state sensor 18 is a body measuring instrument, thephysical features including the weight, the body fat percentage, and thebody composition of the writing person may be extracted. When the statesensor 18 is a microphone, the physical features including thevoiceprint and the voice volume of the writing person may be extracted,and “emotional features” that represent the state of delight, anger,sorrow, and pleasure may be extracted.

In a step S15, the data analyzing section 34 carries out secondarydetermination relating to whether or not a notification to be describedlater is necessary, by using the features calculated in the step S14.For example, the data analyzing section 34 can determine whether or nota notification is necessary depending on whether or not pluralindividual conditions are all satisfied or by converting satisfactionand non-satisfaction of the plural individual conditions to a score.Alternatively, the data analyzing section 34 may carry out thedetermination with higher accuracy by using a determination implement 70in which machine learning has been carried out.

FIG. 5 is a diagram illustrating the configuration of the determinationimplement 70 included in the data analyzing section 34 of FIG. 1. Thedetermination implement 70 is configured of a hierarchical neuralnetwork composed of an input layer 72, intermediate layers 74, and anoutput layer 76, for example. The calculation rule of the determinationimplement 70 is defined based on values of a learning parameter groupthat is an aggregation of learning parameters. For example, the learningparameter group may include coefficients that describe an activationfunction of a unit corresponding to a neuron, weighting factorscorresponding to the strength of a synaptic connection, the numbers ofunits that configure the respective layers, and the number ofintermediate layers 74. The learning parameter group is stored in thestoring section 30 (FIG. 1) in the state in which the respective valuesare settled at the end of the learning, and is timely read out asneeded. The input layer 72 is a layer to which a feature relating to thestate of person Hm or a change in the state is input, and is composed ofplural units. This feature is an input vector including at least onekind of component in “time features” relating to the writing timing,“place features” relating to the writing place, “environmental features”relating to the internal environment of the house 100, “motion features”relating to the motion of the writing person, “emotional features”relating to the emotion of the person, and other features. The outputlayer 76 is a layer that outputs a determination value and that iscomposed of one unit in the example of FIG. 5. The determination valueis an index normalized in a range of [0, 1], for example. A larger valuethereof indicates a determination result indicating that a notificationneed is higher and a smaller value thereof indicates a determinationresult indicating that the notification need is lower.

Although the determination implement 70 is constructed by using theneural network in the above-described example, the method of machinelearning is not limited thereto. For example, various methods includinga logistic regression model, support vector machine (SVM), decisiontree, random forest, and boosting may be used.

In a step S16, the data analyzing section 34 checks whether or not thedetermination result in the step S15 satisfies a secondary determinationcondition. Here, the data analyzing section 34 determines that thedetermination condition is satisfied when the determination value outputfrom the determination implement 70 is larger than a threshold (forexample, 0.7). On the other hand, the data analyzing section 34determines that the determination condition is not satisfied when thedetermination value is equal to or smaller than the threshold. When thedetermination condition is not satisfied (step S16: NO), the operationreturns to the step S11 and the steps S11 to S16 are sequentiallyrepeated until the condition is satisfied. On the other hand, when thedetermination condition is satisfied (step S16: YES), the operationproceeds to the next step S17.

In the step S17, the device control section 36 selects at least onenotification device 58 suitable to the state of person Hm, among one ormore notification devices 58 usable in the house 100. Specifically, thedevice control section 36, by collating the sensor information 42 andthe first device information 44, extracts the ID of the home-use device20 or 22 regarding which the detectable range and the accessible rangepartially overlap (that is, the first device ID). The device controlsection 36 refers to the second device information 46 and extracts theID of one or plural notification devices 58 corresponding to the firstdevice ID (that is, the second device ID).

In a step S18, the device control section 36 instructs the notificationdevice 58 selected in the step S17 to carry out the notificationrelating to the home-use device 20 or 22. Specifically, the devicecontrol section 36 sends out a command signal to instruct thenotification toward the notification device 58 that is the transmissiondestination. The notification device 58 receives the command signal fromthe digital ink server 12 via the network NW and the IoT hub 16.

In a step S19, the notification device 58 notifies the existence of thehome-use device 20 or 22 or the detectable range of the handwritingsensor 52 to person Hm who is present nearby, using the output formaccording to the output function of the notification device.

FIG. 6A to FIG. 6C are diagrams illustrating one example of notificationby the home-use device 22. All diagrams are partial side views of thebathroom 104 illustrated in FIG. 2, and illustrate the states before andafter person Hm steps on a bathroom scale (state sensor 18) that isplaced on a floor 106.

As illustrated in FIG. 6A, before person Hm steps on the bathroom scale,a display (indicator 56, which may coincide with the handwriting sensor52 in FIG. 6A) buried in a wall (home-use device 22) is turned off. Inthis case, the primary determination condition (step S13 in FIG. 4) isnot satisfied and therefore the steps S11 to S13 are repeatedly carriedout.

Then, as illustrated in FIG. 6B, when person Hm steps on the bathroomscale, the primary and secondary determination conditions (steps S13 andS16 in FIG. 4) are both satisfied and the display on the wall startsemitting light from the entire screen. Person Hm can determine thedetectable range of the handwriting sensor 52 by visually observing thelight emission.

Then, as illustrated in FIG. 6C, after a predetermined time has elapsedfrom the start of the light emission by the display, the display on thewall is turned off although person Hm is still on the bathroom scale.This reduces the power consumption compared with the case in which thelight emission is continued.

Thereafter, the operation returns to the step S11 in FIG. 4, and theoperation according to the flowchart is repeatedly carried out tosuccessively execute the first operation of the home UI system 10.

The digital ink server 12 may carry out, concurrently with the firstoperation, reinforcement learning for the determination implement 70(FIG. 5) that carries out the secondary determination. Example of thelearning scheme for reinforcement include: [1] learning in which“whether or not a notification is made” is used as the action selectionand “the existence of handwriting input” is used as the reward, [2]learning in which “plural notification forms” are used as the actionselection and “the existence of reaction by person Hm” is used as thereward, [3] learning in which “plural contents of notification” are usedas the action selection and “the existence of reaction by person Hm” isused as the reward, [4] learning in which “plural home-use devices 22”are used as the action selection and “that a selection has been made” isused as the reward, and so forth.

Similarly, the digital ink server 12 may carry out similar learning(that is, cleansing processing of conditions) also for the primarydetermination. For example, in the case of carrying out the primarydetermination depending on whether or not plural individual conditionsare all satisfied, correction, deletion, and addition of the individualconditions may be carried out in such a manner that the overallpercentage of correct answers becomes higher. Alternatively, in the caseof carrying out the primary determination by converting the satisfactionand non-satisfaction of the plural individual conditions to a score,weighting factors of the individual conditions may be adjusted in such amanner that the overall percentage of correct answers becomes higher.

Summary of First Operation

As described above, the home UI system 10 that manages digital inkincludes the plural state sensors 18 capable of detecting a state in thehouse 100 or a change in the state, and the plural home-use devices 20and 22 that are arranged in the house 100 or that form part of the house100 and that each include the handwriting sensor 52 capable of detectinghandwriting by person Hm. The home UI system 10 also includes one ormore notification devices 58 (notification units) that notify theexistence of the home-use device 20 or 22 or the detectable region ofthe handwriting sensor 52 to person Hm, and the device control section36 (controller) that instructs at least one notification device 58 tocarry out the notification when it is determined that the notificationis necessary from a detection result of one or more state sensors 18.

In an notification method and an notification program that use the homeUI system 10, the digital ink server 12 as a computer carries out aninstruction step (S18 in FIG. 4) of instructing at least onenotification device 58 to carry out the notification when it isdetermined that the notification is necessary from a detection result ofone or more state sensors 18. This enables timely notification to personHm, taking into account the state, or a change in the state, of personHm who is highly likely to use of the home-use device 20 or 22. Thisavoids creating the inconvenience (here, a loss of an opportunity foruse) that may otherwise arise due to multiple home-use devices 20 and 22coexisting at the time of digital ink input.

The data analyzing section 34 (determining part) may determine whetheror not a notification is necessary by using the determination implement70 that uses at least one feature indicating a detection result of thestate sensor 18 on the input side and that uses the determination valueindicating whether or not a notification is necessary on the outputside. The device control section 36 may instruct the notificationaccording to the determination result by use of the determinationimplement 70. The determination implement 70 may be configured to allowthe learning parameter group defining the calculation rule to be updatedby reinforcement learning.

Second Operation

For example, a situation is contemplated in which plural members of thehouse 100 share one electronic pen 54. In this case, it is not possibleto use identification information (hereinafter, pen ID) of theelectronic pen 54 to identify the actual writing person. Thus, the homeUI system 10 may be configured to perform the “second operation” toestimate the writing person without a separate input operation. Thesecond operation will be described below with reference to a flowchartof FIG. 7 and FIG. 8.

Flow of Second Operation

In a step S21 in FIG. 7, the data acquiring section 32 of the digitalink server 12 checks whether or not data that indicates handwriting byperson Hm (hereinafter, referred to also as “handwriting data”) has beenacquired from any one of the plural home-use devices 20 and 22 arrangedin the house 100. When the handwriting data has not yet been acquired(step S21: NO), the operation stays in the step S21 until thehandwriting data is acquired. On the other hand, when the handwritingdata has been acquired (step S21: YES), the operation proceeds to thenext step S22.

In the step S22, the data acquiring section 32 acquires state data ofthe home-use device 20 or 22 that is the acquisition source of thehandwriting data acquired in the step S21. Specifically, the dataacquiring section 32 collates the sensor information 42 of FIG. 3A andthe first device information 44 of FIG. 3B and acquires the state datafrom one or more state sensors 18 regarding which the detectable rangeand the accessible range partly overlap.

In a step S23, the data analyzing section 34 calculates features fromthe state data acquired in the step S22. This calculation is anoperation similar to the calculation made in the step S14 in FIG. 4, andthus specific description thereof is omitted.

In a step S24, the data analyzing section 34 estimates the writingperson who has inputted handwriting by using the features calculated inthe step S23. For example, when a reference value of the feature isstored regarding each member of the house 100, the data analyzingsection 34 quantitatively evaluates the degree of similarity of thefeature regarding each member, and estimates that the member with themaximum evaluation value is the writing person. Alternatively, the dataanalyzing section 34 may carry out estimation with higher accuracy byusing a discrimination implement 80, in which machine learning has beencarried out.

FIG. 8 is a diagram illustrating the configuration of the discriminationimplement 80 included in the data analyzing section 34 of FIG. 1. Thediscrimination implement 80 is configured by a hierarchical neuralnetwork composed of an input layer 82, intermediate layers 84, and anoutput layer 86, for example. The definition of the learning parametergroup may be the same as with the determination implement 70 of FIG. 5or may be different. Although the discrimination implement 80 isconstructed by using the neural network in the above-described example,the method of machine learning is not limited thereto.

The input layer 82 is a layer to which a feature relating to the statein which writing has been carried out is input, and is composed ofplural units. This feature is an input vector including at least onekind of component in “handwriting features” relating to the form ofhandwriting, “time features” relating to the writing timing, “placefeatures” relating to the writing places, “physical features” relatingto the body of the writing person, “motion features” relating to themotion of the writing person, and other features.

The output layer 86 is a layer that outputs a group of labels of membersof the house 100, and is composed of five units in the example of FIG.8. The group of the labels is an output vector having label values thatrepresent the likelihood of family members 1 to 5 as five components.For example, it is estimated that the writing person is “family member1” when the label value of “family member 1” is the largest in theoutput label group regarding the label values normalized to a range of[0, 1].

In a step S25, the digital ink server 12 supplies the handwriting dataacquired in the step S21 to the storage device 14 together withidentification information of the writing person estimated in the stepS24 (that is, user ID). Therefore, the handwriting data is accumulatedin the handwriting information DB 38 in the state of being associatedwith the proper user ID.

Thereafter, the operation returns to the step S21 in FIG. 7, and theoperation according to the flowchart is repeatedly carried out tosuccessively execute the second operation of the home UI system 10.

Effects of Second Operation

As described above, the home UI system 10 that manages digital inkincludes the plural state sensors 18 capable of detecting a state in thehouse 100 or a change in the state, and the home-use device 20 or 22that is arranged in the house 100 or that forms part of the house 100and that includes the handwriting sensor 52 capable of detectinghandwriting by person Hm. The home UI system 10 also includes the dataanalyzing section 34 (estimation unit) that, when an input of thehandwriting is detected by the home-use device 20 or 22, estimates thewriting person who has inputted the handwriting by using a detectionresult of one or more of the state sensors 18.

In an estimation method and an estimation program that use the home UIsystem 10, the digital ink server 12 as a computer carries out anestimation step (S24 in FIG. 7) of estimating, when an input of thehandwriting is detected by the home-use device 20 or 22, the writingperson who has inputted the handwriting by using a detection result ofone or more of the state sensors 18. This allows for high-accuracyestimation of the wiring person taking into account the state of thewriting person or a change in the state. This avoids creating theinconvenience (here, a mistake in associating data) that may otherwisearise due to multiple writing persons coexist at the time of digital inkinput.

Third Operation

For example, a use case is considered in which the same person Hm writesdown multiple ideas as they come to mind while living in the house 100,each time on the nearest home-use device 20 or 22. However, if multiplepieces of information are fragmentally accumulated, this may not lead toany new creation. Thus, the home UI system 10 may be configured toperform the “third operation” to more effectively present the writteninformation that has been written down by person Hm in the past. Thethird operation will be described below with reference to a flowchart ofFIG. 9 and FIG. 10A and FIG. 10B.

Flow of Third Operation

In a step S31 in FIG. 9, the control section 28 of the digital inkserver 12 checks whether or not a predetermined operation is receivedfrom (or on) the terminal device 24. The “predetermined operation” maybe, for example, an operation of activating a viewing application ofdigital ink and tapping a button on a viewing screen. When thepredetermined operation has not been received (step S31: NO), theoperation stays in the step S31 until the predetermined operation isreceived. On the other hand, when the predetermined operation has beenreceived (step S31: YES), the third operation proceeds to the next stepS32.

In the step S32, the data acquiring section 32 reads out desiredhandwriting data from the handwriting information DB 38. Prior to thisreading-out, the data acquiring section 32 identifies the second deviceID of the terminal device 24 by referring to the second deviceinformation 46 and acquires one or more first device IDs associated withthe second device ID. Thereafter, the data acquiring section 32 readsout, from the handwriting information DB 38, the user ID of the owner ofthe terminal device 24 and at least part of handwriting data associatedwith the first device ID. In the case of partially reading out thehandwriting data, the data acquiring section 32 may impose a searchcondition relating to the creation time, creation place, and so forth ofthe handwriting data, or may extract a predetermined ratio (amount) ofthe handwriting data at random.

In a step S33, the device control section 36 identifies past writteninformation that is a display target from the handwriting data read outin the step S32. Here, the “written information” is written-by-handinformation indicated by handwriting, or content information that isspecified by handwriting within electronic content such as within anelectronic book. For example, the former handwriting corresponds to awritten-by-hand note or an annotated comment, and the latter handwritingcorresponds to an annotation including an underline and an enclosingline.

In a step S34, the device control section 36 instructs the terminaldevice 24, from which the predetermined operation has been received inthe step S31, to display the written information identified in the stepS33. Specifically, the device control section 36 sends display data usedto display the written information toward the terminal device 24 that isthe transmission destination. The terminal device 24 receives thedisplay data from the digital ink server 12 via the network NW and theIoT hub 16.

In a step S35, the terminal device 24 displays the written informationon a display screen by using the display data supplied together with theinstruction in the step S34. The terminal device 24 may display pluralpieces of written information, in turn, sequentially in time-seriesorder, or may display plural pieces of written information, in turn, inrandom order. In the displaying, various display forms may be used (forexample, size of characters, color, position, orientation, whether ornot display of a time stamp exists, and so forth), and various controlmethods may be used (for example, display time, display switching cycle,visual effects, and so forth).

FIG. 10A and FIG. 10B are diagrams each illustrating one example ofdisplay on the terminal device 24. It is assumed that the terminaldevice 24 displays plural pieces of written information, in turn,including: (A) written-by-hand information 111 at a writing timing T1,and (B) content information 112 at a writing timing T2, which aredisplayed in turn. The writing timing T1 may be a timing considerablyearlier than the writing timing T2 (for example, several years prior).

The written-by-hand information 111 of FIG. 10A represents a noterelating to the applicability of a touch sensor. Specifically, at thewriting timing T1, it is determined that a touch sensor composed ofindium tin oxide (ITO) can be applied to/in the wall, whereasapplication to the “window” is technically difficult and application tothe “floor” is meaningless. The content information 112 of FIG. 10Bindicates a technological trend relating to the touch sensor.Specifically, at the writing timing T2, it has been suggested thatforming a touch sensor using a metal mesh pattern is highly feasible.

Person Hm, by visually recognizing the display, realizes that it istechnically feasible to incorporate a touch sensor into a window, whichleads to creation of a “home UI system” as a new business model, inwhich a “house itself” functions as an input-output device of digitalink.

After the step S35 in FIG. 9, the third operation returns to the stepS31, and the operation according to the flowchart is repeatedly carriedout to successively execute the third operation of the home UI system10.

Effects of Third Operation

As described above, the home UI system 10 that manages digital inkincludes the plural home-use devices 20 and 22 that are arranged in thehouse 100 or that form part of the house 100 and that each include thehandwriting sensor 52 capable of inputting handwriting by person Hm. Thehome UI system 10 also includes the storage device 14 that storeshandwriting data that indicates the form of handwriting in such a mannerthat the handwriting data is associated with the home-use device 20 or22. The home UI system 10 also includes the terminal device 24 that isconfigured to display an image or video and is associated with one ormore home-use devices 20 and 22. The home UI system 10 includes thedevice control section 36 (controller) that, when receiving apredetermined operation through the terminal device 24, acquires fromthe storage device 14 handwriting data of the home-use devices 20 and 22corresponding to the terminal device 24, and that instructs the terminaldevice 24 to display written-by-hand information indicated by theacquired handwriting data or content information specified by theacquired handwriting data.

In a display method and a display program that use the home UI system10, the digital ink server 12 as a computer carries out an instructionstep (S34 in FIG. 9) of acquiring, when receiving a predeterminedoperation through the terminal device 24, handwriting data of thehome-use devices 20 and 22 corresponding to the terminal device 24 fromthe storage device 14, and instructing the terminal device 24 to displaywritten-by-hand information indicated by the acquired handwriting dataor content information specified by the acquired handwriting data. Thismakes it possible to display previously written information input to thehome-use devices 20 and 22 as an image or video in a consolidated manneron the specific terminal device 24, and to effectively outputhandwriting information indicated by the digital ink.

In particular, when the terminal device 24 displays plural pieces ofhandwritten information or plural pieces of content information in turnat random, it becomes possible to successively present combinations ofpieces of fragmented information that person Hm may not think of. Thus,organic linkage of pieces of information in various perspectives may bereadily formed in the brain of person Hm, which leads to furthercreative activity of person Hm.

Fourth Operation

Person Hm, by looking back on memories in the past, may obtain physicaland mental effects such as psychological comfort, activation of thebrain, and a feeling of belonging to home. The home UI system 10 may beconfigured to perform the “fourth operation” to trigger recollection ofthe content of writing in the past. The fourth operation will bedescribed below with reference to a flowchart of FIG. 11 and FIG. 12A toFIG. 13C.

Flow of Fourth Operation

In a step S41 in FIG. 11, the data acquiring section 32 of the digitalink server 12 acquires state data from one or plural state sensors 18.This acquisition is an operation similar to the acquisition made in thestep S11 in FIG. 4, and thus specific description thereof is omitted.

In a step S42, the data analyzing section 34 carries out primarydetermination relating to whether or not a notification is necessary byusing the state data acquired in the step S41. Specifically, the dataanalyzing section 34 analyzes the state data from multiple perspectivesor in a time-series manner by using various analytical methods includingsensor fusion, and checks whether or not a predetermined state of personHm has been detected. Here, the “predetermined state” may mean, forexample, the state in which person Hm relaxes near the home-use device20 or 22. Examples of such state include: [1] the state in which personHm sits in a chair or sofa, [2] the state in which person Hm standsstill, [3] the state in which person Hm is viewing the specific home-usedevice 20 or 22, and so forth.

In a step S43, the data analyzing section 34 checks whether or not thedetermination result in the step S42 satisfies a primary determinationcondition. When the determination condition is not satisfied (step S43:NO), the operation returns to the step S41 and the steps S41 to S43 aresequentially repeated until the condition is satisfied. On the otherhand, when the determination condition is satisfied (step S43: YES), theoperation proceeds to the next step S44.

In the step S44, the data analyzing section 34 calculates pluralfeatures used for secondary determination from the state data acquiredin the step S41 or state data additionally acquired. The data analyzingsection 34 may calculate the same features as in the case of the stepS14 in FIG. 4 or may calculate different features.

In a step S45, the data analyzing section 34 carries out secondarydetermination relating to whether or not displaying to be describedlater is necessary, by using the features calculated in the step S44.For example, the data analyzing section 34 can determine whether or notdisplaying is necessary depending on whether or not plural individualconditions are all satisfied or through converting satisfaction andnon-satisfaction of the plural individual conditions to a score.Alternatively, the data analyzing section 34 may carry out thedetermination with higher accuracy by using a determination implement 90in which machine learning has been carried out. This determinationimplement 90 can employ a configuration similar to that of thedetermination implement 70 illustrated in FIG. 5.

In a step S46, the data analyzing section 34 checks whether or not thedetermination result in the step S45 satisfies a secondary determinationcondition. Here, the data analyzing section 34 determines that thedetermination condition is satisfied when a determination value outputfrom the determination implement 90 is larger than a threshold.

On the other hand, the data analyzing section 34 determines that thedetermination condition is not satisfied when the determination value isequal to or smaller than the threshold. When the determination conditionis not satisfied (step S46: NO), the operation returns to the step S41,and the steps S41 to S46 are sequentially repeated until the conditionis satisfied. On the other hand, when the determination condition issatisfied (step S46: YES), the operation proceeds to the next step S47.

In the step S47, the data analyzing section 34 selects, based on theanalytical result in the step S42, one home-use device 22 that ispresent near person Hm and that has a touch panel display.Alternatively, the data analyzing section 34 may select the home-usedevice 22 through collation of the sensor information 42 and the firstdevice information 44 similarly to the case of the step S17 (FIG. 4).

In a step S48, the data acquiring section 32 reads out handwriting dataof the home-use device 22 selected in the step S47 from the handwritinginformation DB 38. Written-by-hand information indicated by handwritingis included in the handwriting data. In the case of partially readingout the handwriting data, the data acquiring section 32 may impose asearch condition relating to the creation time, creation place, and soforth of the handwriting data, or may extract a predetermined ratio(amount) of the handwriting data at random.

In a step S49, the device control section 36 sends out display dataincluding the handwriting data read out in the step S48 toward thehome-use device 22 selected in the step S47. The home-use device 22receives the display data from the digital ink server 12 via the networkNW and the IoT hub 16.

In a step S50, the home-use device 22 displays written information on adisplay screen by using the display data supplied in the step S49.Similarly to the case of the step S35 (FIG. 9), the home-use device 22may display plural pieces of written information, in turn, intime-series order or may display plural pieces of written information,in turn, in random order. In the displaying, various display forms maybe used (for example, size of characters, color, position, orientation,whether or not display of a time stamp exists, whether or not display ofthe writing person exists, and so forth), and various control methodsmay be used (for example, display time, switching cycle, visual effects,and so forth).

FIG. 12A to FIG. 12C are diagrams illustrating a first example of thedisplay by the home-use device 22. Here, the home-use device 22 is atablet with which an electronic book can be viewed, and which candisplay plural pieces of written information including pieces of contentinformation 121, 122, and 123 extracted at random, in turn. The contentinformation 121 of FIG. 12A indicates an annotation written to a part ofa nursery rhyme. The content information 122 of FIG. 12B indicates anannotation written to a part of a fairy tale. FIG. 12C illustrates anannotation written to a mathematical diagram.

FIG. 13A to FIG. 13C are diagrams illustrating a second example of thedisplay by the home-use device 22. Here, the home-use device 22 is atable including a touch panel display and is configured to displayplural pieces of written information including pieces of written-by-handinformation 124, 125, and 126 extracted at random, in turn. Thewritten-by-hand information 124 of FIG. 13A is a shopping list writtenby a “mother.” The written-by-hand information 125 of FIG. 13B is anillustration written by a “father.” FIG. 13C is a message written by a“child.”

The home-use device 22 carries out displaying various content, in turn,at a constant slow cycle as with the slow cycle of a pendulum clock.Person Hm, by visually recognizing the display which does not appear tohave any special meaning, may become attached to the home-use device 22and be triggered to remember memories of the house 100. Person Hm mayfeel the sense of belonging to the house 100.

After the step S50 in FIG. 11, the operation returns to the step S41,and the operation according to the flowchart is repeatedly carried outto successively execute the fourth operation of the home UI system 10.

The digital ink server 12 may carry out, concurrently with the fourthoperation, reinforcement learning for the determination implement 90(FIG. 5) that carries out the secondary determination. The reinforcementlearning may be carried out based on a learning scheme in which “whetheror not displaying is made” is used as the action selection and “theexistence of reaction by person Hm” is used as the reward, for example.The digital ink server 12 may carry out similar learning (that is,cleansing processing of conditions) also for the primary determination.

Effects of Fourth Operation

As described above, the home UI system 10 that manages digital inkincludes one or plural state sensors 18 capable of detecting a state inthe house 100 or a change in the state, and the home-use device 22 thatis arranged in the house 100 or that forms part of the house 100 andthat includes a touch panel display capable of inputting and outputtinghandwriting by person Hm. The home UI system 10 also includes thestorage device 14 that stores handwriting data that indicates the formof handwriting in a manner such that the handwriting data is associatedwith the home-use device 22. The home UI system 10 includes the devicecontrol section 36 (controller) that, when it is determined thatdisplaying is necessary based on a detection result of one or more ofthe state sensors 18, acquires from the storage device 14 handwritingdata corresponding to the home-use device 22 located near person Hm, andinstructs the home-use device 22 to display written-by-hand informationindicated by the acquired handwriting data.

In the display method and the display program that use the home UIsystem 10, the digital ink server 12 as a computer carries out, when itis determined that displaying is necessary from a detection result ofone or more of the state sensors 18, acquiring from the storage device14 handwriting data corresponding to the home-use device 22 located nearperson Hm and carrying out the instruction step (S49 in FIG. 11) ofinstructing the home-use device 22 to display written-by-handinformation indicated by the acquired handwriting data. This allows fortimely displaying of content for person Hm, taking into account thestate, or a change in the state, of person Hm who indicates a highinterest in the home-use device 22, and makes it possible to effectivelyoutput written information previously input from the home-use device 22as an image or video.

In particular, when the home-use device 22 displays plural pieces ofwritten-by-hand information, in turn, at random, it becomes possible forperson Hm to successively look back on memories of the past, which wouldtrigger physical and mental effects such as psychological comfort andactivation of the brain. Furthermore, by carrying out display in a modethat seems not to have a special meaning, person Hm may experience thesense of being present and belonging in the house 100 to remembermemories of the house 100. The data analyzing section 34 (determinationunit) may determine whether or not displaying is necessary by using thedetermination implement 90 that uses at least one feature indicating adetection result of the state sensor 18 on the input side and that usesthe determination value indicating whether or not displaying isnecessary on the output side. The device control section 36 may instructdisplaying according to the determination result by use of thedetermination implement 90. This determination implement 90 may beconfigured to allow the learning parameter group defining thecalculation rule to be updated by reinforcement learning.

Modification Examples

The present disclosure is not limited to the above-described embodimentsand various modifications are possible. Further, various configurationsmay be selectively combined as long as technical contradiction is notcreated.

Regarding the step S15 (FIG. 4) of the first operation or the step S45(FIG. 11) of the fourth operation, the data analyzing section 34 maydetermine whether or not a notification or displaying is necessary byusing a feature that indicates the degree of affinity between thehome-use device 20 or 22 and the house 100. This feature may becalculated based on an image taken by a camera, or may be stored inadvance in the first device information 44.

Regarding the steps S12 and S15 (FIG. 4) of the first operation or thesteps S42 and S45 (FIG. 11) of the fourth operation, in the case ofcarrying out the two-stage determination relating to whether or not anotification or displaying is necessary, a device on the downstream side(for example, IoT hub 16 in FIG. 1) may be configured to perform thefunction of the primary determination. This allows distributedprocessing by edge computing, and the load on the digital ink server 12is alleviated.

Furthermore, the data analyzing section 34 may carry out the primarydetermination by using a relatively smaller number of state sensors 18and carry out the secondary determination by using a relatively largernumber of state sensors 18. This can alleviate the communication load ona link with the state sensors 18. Similarly, the data analyzing section34 may carry out the primary determination by using determinationprocessing, in which the amount of calculation per instance ofprocessing is relatively smaller, and carry out the secondarydetermination by using determination processing, in which the amount ofcalculation per instance of processing is relatively larger. This canalleviate the calculation load on the digital ink server 12.

Regarding the step S19 (FIG. 4) of the first operation, the notificationdevice 58 may carry out the notifications, plural times at differenttimes. For example, the notification device 58 may carry out a firstnotification in an output form having relatively lower directivity (forexample, sound) and thereafter carry out a second notification in anoutput form having relatively higher directivity (for example, light).

Regarding the step S48 (FIG. 11) of the fourth operation, the dataacquiring section 32 may limit handwriting data that is read out fromthe handwriting information DB 38 according to a disclosure limitationset in advance. This allows for protecting private information thatshould not be shown to anyone even at home, to thereby increase theutility of the home UI system 10.

The various embodiments described above can be combined to providefurther embodiments. All of the U.S. patents, U.S. patent applicationpublications, U.S. patent applications, foreign patents, foreign patentapplications and non-patent publications referred to in thisspecification and/or listed in the Application Data Sheet areincorporated herein by reference, in their entirety. Aspects of theembodiments can be modified, if necessary to employ concepts of thevarious patents, applications and publications to provide yet furtherembodiments.

1. A home user interface system that manages digital ink, the home userinterface system comprising: a plurality of state sensors capable ofdetecting a state in a house or a change in the state; a plurality ofhome-use devices that are provided in the house or that form part of thehouse and that each include a handwriting sensor capable of detectinghandwriting by a person; one or more notification units configured toperform a notification to notify the person of existence of the home-usedevice or a detectable region of the handwriting sensor; and acontroller which, responsive to a determination based on a detectionresult from one or more of the plurality of state sensors that thenotification is necessary, instructs at least one of the notificationunits to carry out the notification.
 2. The home user interface systemaccording to claim 1, comprising: a determination unit that determineswhether or not the notification is necessary, by using a determinationimplement that uses at least one feature indicating a detection resultfrom the state sensor on an input side and that uses a determinationvalue indicating whether or not the notification is necessary on anoutput side, wherein the controller instructs the notification accordingto a determination result by use of the determination implement.
 3. Thehome user interface system according to claim 2, wherein thedetermination implement is configured to allow a learning parametergroup defining a calculation rule to be updated by reinforcementlearning.
 4. A home user interface system that manages digital ink, thehome user interface system comprising: a plurality of state sensorscapable of detecting a state in a house or a change in the state; ahome-use device that is provided in the house or that forms part of thehouse and that includes a handwriting sensor capable of detectinghandwriting by a person; and an estimation unit which, responsive to adetection by the home-use device of a handwriting input, estimates aperson who has inputted the handwriting based on a detection result fromone or more of the plurality of state sensors.
 5. A home user interfacesystem that manages digital ink, the home user interface systemcomprising: a plurality of home-use devices that are provided in a houseor that form part of the house and that each include a handwritingsensor capable of inputting handwriting by a person; a storage devicethat stores handwriting data indicating a form of handwriting in such amanner as to associate the handwriting data with the home-use device; aterminal device configured to display an image or a video and associatedwith one or more of the plurality of home-use devices; and a controllerthat, in response to receiving a predetermined operation through theterminal device, acquires handwriting data of the home-use deviceassociated with the terminal device from the storage device, andinstructs the terminal device to display written-by-hand informationindicated by the handwriting data that has been acquired or contentinformation specified by the handwriting data that has been acquired. 6.The home user interface system according to claim 5, wherein theterminal device displays plural pieces of the written-by-handinformation or plural pieces of the content information, in turn, atrandom.
 7. A home user interface system that manages digital ink, thehome user interface system comprising: a plurality of state sensorscapable of detecting a state in a house or a change in the state; ahome-use device that is provided in the house or that forms part of thehouse and that includes a touch panel display capable of inputting andoutputting handwriting by a person; a storage device that storeshandwriting data indicating a form of handwriting in such a manner as toassociate the handwriting data with the home-use device; and acontroller that, in response to a determination based on a detectionresult from one or more of the plurality of state sensors thatdisplaying is necessary, acquires handwriting data associated with thehome-use device located near the person from the storage device, andinstructs the home-use device to display written-by-hand informationindicated by the handwriting data that has been acquired.
 8. The homeuser interface system according to claim 7, wherein the home-use devicedisplays the plural pieces of written-by-hand information, in turn, atrandom.
 9. The home user interface system according to claim 7,comprising: a determination unit that determines whether or not thedisplaying is necessary, by using a determination implement that uses atleast one feature indicating a detection result from the state sensor onan input side and that uses a determination value indicating whether ornot the displaying is necessary on an output side, wherein thecontroller instructs the displaying according to a determination resultby use of the determination implement.
 10. The home user interfacesystem according to claim 9, wherein the determination implement isconfigured to allow a learning parameter group defining a calculationrule to be updated by reinforcement learning.